config: add entry-point-based plugin schema discovery#10994
Open
adamlabadorf wants to merge 5 commits intotreeverse:mainfrom
Open
config: add entry-point-based plugin schema discovery#10994adamlabadorf wants to merge 5 commits intotreeverse:mainfrom
adamlabadorf wants to merge 5 commits intotreeverse:mainfrom
Conversation
Allow DVC filesystem plugins to declare a REMOTE_CONFIG class attribute on their filesystem class. At import time, _discover_plugin_schemas() iterates over installed dvc.fs entry points and merges any declared config schemas into REMOTE_SCHEMAS, enabling ByUrl to accept custom URL schemes without changes to DVC core. This makes DVC truly extensible for third-party storage backends (e.g. OSF, GitLab packages) that define their own URL schemes. Existing hardcoded schemes are never overwritten. Relates to treeverse#9711.
|
|
adamlabadorf
force-pushed
the
feature/plugin-schema-discovery
branch
from
1029b4f to
81944fe
Compare
Allow DVC filesystem plugins to declare a REMOTE_CONFIG class attribute on their filesystem class. At import time, _discover_plugin_schemas() iterates over installed dvc.fs entry points and merges any declared config schemas into REMOTE_SCHEMAS, enabling ByUrl to accept custom URL schemes without changes to DVC core. This makes DVC truly extensible for third-party storage backends (e.g. OSF, GitLab packages) that define their own URL schemes. Existing hardcoded schemes are never overwritten. Fixes treeverse#10993
adamlabadorf
force-pushed
the
feature/plugin-schema-discovery
branch
from
81944fe to
e244e12
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10994 +/- ##
==========================================
+ Coverage 90.68% 91.00% +0.32%
==========================================
Files 504 506 +2
Lines 39795 41213 +1418
Branches 3141 3263 +122
==========================================
+ Hits 36087 37506 +1419
- Misses 3042 3069 +27
+ Partials 666 638 -28 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
entry_points(group=...) keyword argument was added in Python 3.10. On Python 3.9, use the dict-based entry_points().get(group, []) API instead. Fixes TypeError on Python 3.9: TypeError: entry_points() got an unexpected keyword argument 'group'
Extract entry point fetching into _get_dvc_fs_entry_points() helper so that tests can mock it directly rather than mocking entry_points(). This avoids the Python 3.9 vs 3.10+ dict/list API difference leaking into test mocks.
mypy evaluates sys.version_info >= (3, 10) as always True on 3.10+ environments, flagging the 3.9 fallback return as unreachable. Extend the type: ignore comment to cover both call-arg and unreachable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10993
Summary
Adds
_discover_plugin_schemas()todvc/config_schema.py, which iterates over installeddvc.fsentry points at import time and merges anyREMOTE_CONFIGclass attributes intoREMOTE_SCHEMAS. This lets third-party filesystem plugins register their own URL schemes and config keys without modifying DVC core.Changes
dvc/config_schema.py— adds_discover_plugin_schemas()(~25 lines) called once at module loadtests/unit/test_plugin_schema_discovery.py— 7 new unit testsBehavior
Plugin declares its config schema:
DVC discovers it automatically when the plugin is installed — no changes to DVC core needed per-plugin.
Properties
s3,gs, etc.) are never overwrittenprotocolvaluestest_plugin_schema_discovery+test_config)Motivation
Developed alongside dvc-osf, a plugin for Open Science Framework storage. Without this change,
dvc remote add myremote osf://...raisesUnsupported URL type osf://before the filesystem ever loads. With it, all DVC remote operations work correctly (30 integration tests passing against a real OSF project).